Annotations in software applications for invoking dialog system functions

ABSTRACT

A method for expanding software application functionalities processing is disclosed herein. The method comprises receiving a user request within a software application. The software application is enhanced with annotations, which include metadata. The method continues with loading the metadata from the annotations to a dialog system located on a remote device, sending at least a part of the user request to the dialog system, and receiving a dialog system response to the user request from the dialog system. At least one of the annotations associated with the dialog system response is identified. A code associated with the identified annotation is invoked within the software application.

BACKGROUND

Currently, dialog systems are widely used in the information technologyindustry, especially in the form of mobile applications for cell phonesand tablet computers. A dialog system can include a computer-based agenthaving a human-centric interface for accessing, processing, managing,and delivering information. Dialog systems are also known as chatinformation systems, spoken dialog systems, conversational agents,chatter robots, chatterbots, chatbots, chat agents, digital personalassistants, automated online assistants, and so forth. All these termsare within the scope of the present disclosure and referred to as a“Dialog System” for simplicity.

Traditionally, a dialog system interacts with its users in naturallanguage to simulate an intelligent conversation and providepersonalized assistance to the users. For example, a user may generaterequests to the dialog system in the form of conversational questions,such as “What is the weather like in Palo Alto?” and receivecorresponding answers from the dialog system in the form of audio and/ordisplayable messages. The users may also provide voice commands to thedialog system requesting the performance of certain functions including,for example, generating e-mails, making phone calls, searchingparticular information, acquiring data, navigating, requestingnotifications or reminders, and so forth. These and otherfunctionalities make dialog systems very popular as they are of greathelp, especially for holders of portable electronic devices such assmart phones, cellular phones, tablet computers, gaming consoles, andthe like.

Although the demand for dialog systems for third party mobileapplications and web services is constantly growing, it is not an easytask to create a well-operating dialog system. Each dialog system has adialog system interface and dialog system engine. The dialog systeminterface is responsible for receiving user inputs and delivering dialogsystem responses to the user. The dialog system engine is responsiblefor transforming voice user inputs into text inputs, interpreting textinputs, and generating corresponding responses to text inputs. Theprocess running on the dialog system engine is also known as naturallanguage processing (NLP). Development of NLP models is not only timeconsuming, but is also a highly technical task. Accordingly, applicationdevelopers may struggle to develop a dialog system for integrating itsfunctionality with a mobile application or web service. Accordingly,there is a need in the art to simplify the process of creating,developing, and maintaining dialog systems for software applications andweb services.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

The present technology provides for an online platform configured tomaintain Dialog System Engines that can process user requests acquiredvia software applications installed on user devices. The presenttechnology also allows the software developers to annotate certaindialog system functions in a source code of their software applicationsin a separate development environment (i.e., without the need to codedialog system functions). The annotations and dialog descriptors, whichare either built into or provided with software application code, wheninvoked, cause Dialog System Interfaces to implement desired dialogsystem functions with respect to certain user inputs and deliver dialogsystem responses to end users. This significantly simplifies the designprocess of software applications, thereby allowing the softwaredevelopers to integrate one or more dialog system functions intosoftware applications without dedicating time and resources fordeveloping and maintaining dialog systems. This also allows obtaining adialog support for software applications deployed in particularapplication environments. For example, iOS and Android applications canreceive a dialog support defined in the application and enabled when theapplication is deployed on a device.

In accordance with various embodiments of the present disclosure,software developers may insert annotations in the form of metadatawithin software source code during design time. Alternatively,developers can use pre-built annotations to insert annotations intocompiled dialog definitions during design time. In yet more embodiments,annotations can be also uploaded or synchronized with an onlineplatform. Further, the metadata is used later at load time to specifywhich dialog system functions will be implemented when the program runs,based on one or more criteria. In this way, the annotations allowdeferring implementations of certain dialog system functions andbuilding dialog systems when the actual software package is alreadycreated.

At runtime, the user device running the Dialog System Interface receivesuser requests. The user requests are used to identify and invokeannotations integrated into the code of software applications. The userdevice, through a particular Dialog System Interface, implements one ormore dialog system functions based on invoked annotations and generatesdialog system responses based on the annotations.

Provided is a method for expanding software application functionalities.The method may commence with receiving a user request within a softwareapplication. The software application may be enhanced with annotations.The annotations may include metadata. The method may further includesending the user request to a dialog system located on a remote device.The dialog system may send a dialog system response to the user request.Upon receiving the dialog system response, at least one of theannotations associated with the dialog system response may beidentified. Based on the identification, a code may be invoked withinthe software application. The code may be associated with the identifiedannotation.

Additional objects, advantages, and novel features will be set forth inpart in the detailed description, which follows, and in part will becomeapparent to those skilled in the art upon examination of the followingdetailed description and the accompanying drawings or may be learned byproduction or operation of the example embodiments. The objects andadvantages of the concepts may be realized and attained by means of themethodologies, instrumentalities, and combinations particularly pointedout in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in thefigures of the accompanying drawings, in which like references indicatesimilar elements and, in which:

FIG. 1 illustrates an environment within which systems and methods forexpanding software application functionalities can be implemented, inaccordance with some embodiments.

FIG. 2 is a block diagram showing various modules of a dialog systemengine, in accordance with certain embodiments.

FIG. 3 is a block diagram of a system for expanding software applicationfunctionalities, in accordance with some example embodiments.

FIG. 4 is a flow chart illustrating a method for expanding softwareapplication functionalities, in accordance with some exampleembodiments.

FIG. 5 shows a process flow diagram for a method of creating customdialog system engines, in accordance with some example embodiments.

FIG. 6 shows the process of manual creating of entities for a dialogsystem rule, in accordance with some example embodiments.

FIG. 7 shows the process of manual creating of intents via the platforminterface, in accordance with some example embodiments.

FIG. 8 shows an example rule for receiving a weather forecast, inaccordance with some example embodiments.

FIG. 9 shows a process of testing dialog system elements by developers,in accordance with some example embodiments.

FIG. 10 is a high-level block diagram illustrating an example userdevice suitable for implementing the methods described herein.

FIG. 11 is a high-level block diagram illustrating an example systemsuitable for implementing the methods described herein.

DETAILED DESCRIPTION

The following detailed description includes references to theaccompanying drawings, which form a part of the detailed description.The drawings show illustrations in accordance with example embodiments.These example embodiments, which are also referred to herein as“examples,” are described in enough detail to enable those skilled inthe art to practice the present subject matter.

The embodiments can be combined, other embodiments can be utilized, orstructural, logical and operational changes can be made withoutdeparting from the scope of what is claimed. The following detaileddescription is, therefore, not to be taken in a limiting sense, and thescope is defined by the appended claims and their equivalents.

Present teachings may be implemented using a variety of technologies.For example, the methods described herein may be implemented in softwareexecuting on a computer system or in hardware utilizing either acombination of microprocessors or other specially designedapplication-specific integrated circuits, programmable logic devices, orvarious combinations thereof. In particular, the methods describedherein may be implemented by a series of computer-executableinstructions residing on a transitory or non-transitory storage mediumsuch as a disk drive or computer-readable medium. It should be notedthat methods disclosed herein can be implemented by a computer (e.g., adesktop computer, tablet computer, laptop computer), game console,handheld gaming device, cellular phone, smart phone, smart televisionsystem, and so forth.

The term “software developer” or simply “developer” refers to one ormore of the following: software developer, mobile application developer,application developer, software engineer, software owner, mobileapplication owner, software manager, mobile application manager, dialogsystem owner, and so forth. A developer develops and/or manages a DialogSystem Engine, Dialog System Interface, and/or software applications.

The term “Dialog System” refers to one or more of the following: chatinformation system, spoken dialog system, conversational agent, chatterrobot, chatterbot, chatbot, chat agent, digital personal assistant,automated online assistant, and so forth. Each Dialog System includes a“Dialog System Interface” and a “Dialog System Engine.” Each of theseelements can be customized by the developer.

The term “Dialog System Interface” refers to a computer-human interface,which is configured to acquire user inputs in the form of audio messagesor text messages, and deliver dialog system responses to the users inthe form of audio messages or displayable messages. In one example, aDialog System Interface may be implemented as a widget designed for orintegrated with, a software application, mobile application, middlewareapplication, firmware application, website, and web service, to providea computer-human interface for acquiring user requests and deliveringdialog system outputs to the users. In this disclosure, Dialog SystemInterfaces can be virtual, meaning that they include one or moreannotations only.

The term “Dialog System Engine” refers to a software applicationconfigured to process user inputs and generate responses thereto. In oneexample, Dialog System Engine refers to a computer-enabled orprocessor-enabled system for supporting an associated Dialog SystemInterface by processing user requests and generating correspondingresponses thereto.

The term “annotation” refers to syntactic metadata that can be added tothe source code of a software application. Annotations can invokeimplementation of certain dialog system functions by Dialog SystemEngines and/or Dialog System Interfaces at runtime. Annotations caninclude or be associated with one or more criteria, parameters,variables, classes, methods, and/or packages. Annotations can alsoinclude or be associated with one or more user inputs such as textinputs or speech inputs. In various embodiments, developers can addannotations in the form of dialog metadata to a source code ofparticular software applications. One example of annotations is Javaannotations, although other means for other programming languages andplatforms can be used. In yet other embodiments, annotations can includecodes that define dialog system specific behaviors with a softwareapplication under development.

The present technology provides for a platform enabling creation ofcustom Dialog System Engines serving as backend services for DialogSystem Interfaces. Dialog System Interfaces can be implemented at leastas a part of various software applications, mobile applications,middleware applications, firmware applications, websites, web services,and so forth. In other words, Dialog System Interfaces are on a clientside (in the user device or in a cloud or web service) and provide acomputer-human interface configured to at least acquire user inputs anddeliver dialog system outputs to the users. Dialog System Engines, onthe other hand, support the Dialog System Interfaces by processing userinputs and generating corresponding responses thereto. Thus, a DialogSystem Engine and Dialog System Interface, when interacting with eachother, form a Dialog System. One may refer to a Dialog System Interfacerunning on or accessed from a user device as a “frontend” userinterface, while a Dialog System Engine, which supports the operation ofsuch Dialog System Interface, can be referred to as a “backend” service.

The platform, according to various embodiments of the presentdisclosure, allows software developers to create custom Dialog SystemEngines that will support frontend Dialog System Interfaces. Forexample, if a software developer wants to integrate Dialog Systemfunctionality into a mobile application as an additional feature, thedeveloper can use the platform to create and deploy a custom DialogSystem Engine and link it with or embed it into the mobile application.The mobile application, in turn, will have only a Dialog SystemInterface. In this example, the Dialog System Interface can be activatedby a user when he interacts with the mobile application. The user canmake inquiries to the Dialog System Interface in the form of voiceinputs or text inputs. Upon receipt of a user inquiry, the Dialog SystemInterface can transfer the user inquiry with little or no pre-processingto the linked custom Dialog System Engine, which was previously createdusing the platform. The Dialog System Engine processes the received userinquiry, interprets it, and generates a response to the inquiry based onpredetermined rules and settings. Predetermined rules and settings canbe either defined by developers (i.e., they include specific dialogscenarios, entities, etc.), or built into the platform (e.g., providedvia embeddable Software Development Kit (SDK) or a cloud solution). Theresponse is then delivered to a mobile application for further visual oraudio presentation to the user. In some embodiments, the response mayinclude a response text to be delivered to the user and/or metadata withinstructions for the user device to perform an action (e.g., open abrowser, access certain data online, run a particular application,etc.). In some embodiments, the response may include data (e.g., in thecase of a web service call). In other embodiments, the response mayinclude a callback Uniform Resource Locator (URL) that the Dialog SystemInterface or user device needs to access to obtain a response textand/or metadata.

According to various embodiments of this disclosure, interaction betweena Dialog System Interface and Dialog System Engines can be implementedwith the help of annotations. Software developers can insert custom orpredetermined annotations into a code of a software application suchthat, at runtime, the annotations invoke implementation of certaindialog system functions by Dialog System Engines. This may includetransmittance of user inputs, various parameters, variables, metadata,and, optionally, other data by the software application to a DialogSystem Engine. In response to an invoking request caused by anannotation, the Dialog System Engine processes the user input andoptionally other data, generates a dialog system response, and deliversthe same to the software application. Accordingly, software codingbecomes a much easier task for software developers when the code isexpanded by adding annotations invoking various dialog systemfunctionalities.

The methods of this disclosure enable developers to specify dialogsystem elements in the form of annotations right in a code during adesign phase, instead of defining the dialog system elements separately.This also allow deferring implementation of certain dialog systemfunctions until the runtime of software applications by invokingannotations integrated into code of software applications.

The benefits of methods of this disclosure can be evident from thefollowing example. Assume a software developer creates a new mobileapplication having an interface with a number of buttons, each of whichimplements certain functions. One of the buttons causes the activationof a built-in camera, makes a picture, and stores the same in a memory.This functionality can be specified in a software code using aprogramming language. If the developer needs to enable activation of thecamera and making the picture in response to a user's speech input, suchas “Shoot,” “Take a picture,” “Smile” (i.e., without pressing thebutton), the developer needs to code a dialog system procedure, which isa complex task to accomplish. The present technology allows thedevelopers in these instances to introduce custom or predeterminedannotations into a source code of software applications to invokeimplementation of dialog system functions by certain Dialog SystemInterfaces. In the above example, the developer simply needs to insert apredetermined annotation, such as @CameraAction, into the software codeto expand functionalities of the mobile application by adding thedesired dialog system procedure. In this way, when the user of themobile application says “Shoot,” “Take a picture,” “Smile,” or anotherpredetermined word or phrase, it invokes the annotation to implement aparticular Dialog System function. As a result of the processing, theDialog System Engine generates a dialog system response in the form ofat least one processor-implementable instruction that causes the userdevice to activate the camera, make a picture, and store the same in thememory. In some embodiments, annotations includes a full specificationfor an intent rather than a link to the intent or the intent'sidentifier such as “aCameraAction”. When an annotation includes the fullspecification, a conversational agent can be dynamically created for themobile application. In other embodiments, annotations may include morecomplex dialog system definitions available in an online platform. Forexample, annotations can include, but are not limited to, system anddeveloper-defined entities, dialog trees, and fulfillment routines.

In general, Dialog System Interfaces can be integrated or be an integralpart of a wide range of software applications running on a user device,such as a personal computer (PC) or cellular phone, or on a server sothat dialog systems become a part of a website or web service. DialogSystem Engines can be implemented on a server such that theirfunctionalities can be accessible to Dialog System Interfaces over theInternet, cellular networks, or any other communications means.

FIG. 1 shows a high-level block diagram of example system environment100 suitable for practicing the present technologies. As shown on thisfigure, there is provided a platform 110 for creating and maintainingcustom Dialog System Engines. For these ends, the platform 110 includesa platform interface 112 for creating custom Dialog System Engines andbackend service 114 for maintaining and running custom Dialog SystemEngines 120.

Platform interface 112 may include a graphical user interface (GUI)embedded into a webpage and accessible by application developers via theInternet. In other embodiments, however, platform interface 112 may beimplemented as a software application such as a downloadable softwareapplication or any other software, middleware, or firmware running on oraccessible from an electronic device such as a computer. In the exampleshown in FIG. 1, platform interface 112 is realized as a web accessibleGUI, as will be described below. For simplicity, this disclosure islimited to such embodiments where platform interface 112 is aserver-based solution so that it is accessible via the Internet.Regardless of a particular implementation, platform interface 112enables the developers, through a number of GUI tools, to create one ormore custom Dialog System Engines 120.

Still referencing FIG. 1, backend service 114 is responsible formaintaining and running custom Dialog System Engines 120 that arecreated, for example, by or with the help of platform interface 112.Backend service 114 may operate as a web service providing functionalityto custom Dialog Systems by enabling Dialog System Interfaces 130 tointeract with custom Dialog System Engines 120 maintained at backendservice 114 of platform 110.

As discussed above, Dialog System Interfaces 130 can be provided on aclient side 140. Dialog System Interfaces 130 may be as simple as a GUIenabling users to make inquiries, which are then delivered to backendservice 114 for processing by corresponding Dialog System Engines 120,and to receive responses to the inquires generated by Dialog SystemEngines 120. Dialog System Interfaces 130 may be implemented at least asa part of a software application, mobile application, middlewareapplication, or firmware application, web service, website, and soforth. In various embodiments, Dialog System Interfaces 130 include oneor more annotations integrated into software code.

Still referencing FIG. 1, client side 140 may refer to, but is notlimited to, a user device, terminal, computing device (e.g., laptopcomputer, tablet computer, desktop computer), cellular phone, smartphone, gaming console, remote control, multimedia system, smarttelevision device, set-top box, infotainment system, in-vehiclecomputing device, informational kiosk, robot, and so forth. In theseembodiments, Dialog System Interfaces 130 are a part of software,middleware, or firmware installed on such devices.

In additional embodiments, client side 140 may refer to a networked oronline solution such as a server, hosting service, web service, website, cloud service, and so forth. For example, Dialog System Interface130 can be a widget or GUI provided on one or more web pages enablingend users to make inquiries and get responses thereto. This option issuitable for those instances when a developer, for example, wants tointegrate a Dialog System into a website to provide enhanced customerservice.

As can be seen in FIG. 1, the interaction between Dialog SystemInterfaces 130 and corresponding Dialog System Engines 120 is performedvia a communications network 150. Communications network 150 may includeone or more of the Internet, intranet, cellular network, local areanetwork (LAN), wide area network (WAN), IEEE 802.11 based network, andso forth.

FIG. 1 also shows various third party web resources/services 160provided via one or more web servers. These third party webresources/services 160 can provide information of various types toDialog System Engines 120 or Dialog System Interfaces 130 as part of aresponse to a user request. For example, web resources/services 160 mayrefer to email services, weather services, navigation services, and thelike. Accordingly, if a user makes the inquiry “What is the weather liketoday?,” such information may be automatically acquired by Dialog SystemEngine 120 from one or more third party web resources/services 160 andthen integrated into a dialog system response to be delivered to the enduser.

FIG. 2 shows a high-level architecture 200 of an exemplary Dialog SystemEngine 120, according to an example embodiment. It should be noted thateach module of Dialog System Engine 120 or associated architectureincludes hardware components, software components, or a combinationthereof. Dialog System Engine 120 may be embedded or installed in a userdevice or server, or may be presented as a cloud computing module and/ora distributed computing module.

In the embodiment shown, Dialog System Engine 120 includes an AutomaticSpeech Recognizer (ASR) 210 configured to receive and processspeech-based user inputs into a sequence of parameter vectors. ASR 210further converts the sequence of parameter vectors into a recognizedinput (i.e., a textual input having one or more words, phrases, orsentences). ASR 210 includes one or more speech recognizers such as apattern-based speech recognizer, free-dictation recognizer, address bookbased recognizer, dynamically created recognizer, and so forth.

Further, Dialog System Engine 120 includes NLP module 220 forunderstanding spoken language input. Specifically, NLP module 220 maydisassemble and parse the recognized input to produce utterances, whichare then analyzed utilizing, for example, morphological analysis,part-of-speech tagging, shallow parsing, and the like. NLP module 220may then map recognized input or its parts to meaning representations.

Dialog System Engine 120 further includes dialog manager 230, whichcoordinates the activity of all components, controls dialog flows, andcommunicates with external applications, devices, services, orresources. Dialog manager 230 may play many roles, which includediscourse analysis, knowledge database query, and system actionprediction, based on the discourse context. In some embodiments, dialogmanager 230 may contact one or more task managers (not shown) that mayhave knowledge of specific task domains. In some embodiments, dialogmanager 230 may communicate with various computing, logic, or storageresources 240, which may include, for example, a content storage, rulesdatabase, recommendation database, push notification database,electronic address book, email or text agents, dialog history database,disparate knowledge databases, map database, points of interestdatabase, geographical location determiner, clock, wireless networkdetector, search engines, social networking websites, blogging websites,news feeds services, and many more. In some embodiments, computationalor storage resources 240 include one or more web resources/services 160discussed above.

Dialog manager 230 may employ multiple disparate approaches to generateoutputs in response to recognized inputs. Some approaches include theuse of statistical analysis, machine-learning algorithms (e.g., neuralnetworks), heuristic analysis, and so forth. Dialog manager 230 is oneof the central components of Dialog System Engine 120. The major role ofdialog manager 230 is to select the correct system actions based onobserved evidences and inferred dialog states from the results of NLP(e.g., dialog act, user goal, and discourse history). In addition,dialog manager 230 should be able to handle errors when the user inputhas ASR and NLP errors caused by noises or unexpected inputs.

Dialog System Engine 120 may further include output renderer 250 fortransforming the output of dialog manager 230 into a form suitable forproviding to the user. For example, output renderer 250 may employ atext-to-speech engine or may contact a pre-recorded audio database togenerate an audio message corresponding to the output of dialog manager230. In certain embodiments, output renderer 250 may present or cause topresent the output of dialog manager 230 as a text message, an image, ora video message for further displaying on a display screen of the userdevice. In some example embodiments, output renderer 250 can constituteat least a part of Dialog System Interface 130.

Still referring to FIG. 2, Dialog System Engine 120 includes one or moredialog system rules maintained in at least one rule database 260. DialogSystem Engine 120 may also include or be associated with one or morecontext databases 270, which maintain a plurality of context descriptionelements such as lists of terms, keywords, phrases, expressions, contextvariables, and context parameters (e.g., geolocation, system rate, GUI,etc.) associated with one or more dialog system rules. In other words,context databases 270 include information supporting the process ofdetermining conversational or environmental context for particular userrequests.

Dialog System Engine 120 may also include or be associated with one ormore statistics and usage databases 280, which are configured toaggregate statistical or usage information associated with the operationof Dialog System Engine 120 and/or associated Dialog System Interface130 and/or associated mobile or software application. For example,statistics and usage database 280 may accumulate dialog system logs,which can be later used for optimization of dialog system rules, dialogsystem responding schemes, training machine-learning algorithms ifemployed by Dialog System Engine, and so forth.

The process of creating and operating custom Dialog System Engines 120will now be described with reference to FIG. 1 and other drawings. Inparticular, platform interface 112 provides one or more GUIs having anumber of tools enabling developers to create and customize one or moredialog system elements, which serve as a basis for a custom DialogSystem Engine.

According to various embodiments, dialog system elements include“entities” and “intents.” Each entity may refer to a number of objectshaving the same or similar characteristics. In other words, entities arelists of terms and/or keywords defining objects of one class. In oneexample, an entity may refer to a keyword and a set of its synonyms. Inanother example, an entity may refer to a keyword and a set of itsdefinitions. In yet another example, an entity may refer to a list(e.g., a list of cities, list of names, list of titles, list of brands,list of street names, etc.).

In some embodiments, each entity can have a title. For example, oneentity can be titled as “city” and it will contain a list of cities suchas Arlington, Boston, Chicago, and so forth. In other embodiments, anentity can be titled as a keyword and it can contain synonyms and/ordefinitions of this keyword. In one example, the entity called “music”may include the terms of song, singer, singing, musician, and so forth.In another example, the entity called “artist” may include a list ofmusic bands, music ensembles, or music artists. In another example, theentity called “Beatles” may include a list of possible synonyms, such as“The Beatles,” “Beatles,” “Fab Four,” “Liverpool Legends,” “JohnLennon,” and so forth. In yet another example, there can be an entitycalled “Artist” which may include various artist names, artist namesynonyms, music band names, and so forth.

In some embodiments, Dialog System Engines may include a number ofdefault, pre-configured entities and/or intents. These can includecommon types of entities or intents related to such concepts as time,date, location, and the like. For example, when a developer creates anew Dialog System Engine, it may already have a few entities of commontypes such as a “@System.Date” entity. This entity covers linguisticconstructs related to particular dates and may include the followingterms: “today,” “tomorrow,” “next week,” “January 1,” “January 1 of nextyear,” “next Monday,” “the following Monday,” and so forth.

Further, each intent of a Dialog System Rule includes a dialog systeminteraction scheme, which provides a particular relation between atleast one user request and at least one dialog system linguisticresponse or fulfilment response. The dialog system interaction schemecan be represented by a rule based on a relationship between aparticular action and at least one entity. Actions generally relate toformalized software objects such as JSON (JavaScript Object Notation)objects causing at least one processor to generate linguistic orfulfilment responses associated with at least one entity. Accordingly,each intent can be represented as a logical relation between at leastone action and at least one entity object, for example, as follows:

-   -   a) [Action]@[Entity]    -   b) [Action]@[Entities]    -   c) [Actions]@[Entity]    -   d) [Actions]@[Entities]    -   e) Text@[Entity]    -   f) Text@[Entities]    -   g) Text@[Entity] Text    -   h) [Action] Text@[Entity]

The procedures e) through h) mean that particular information in theform of text is provided with respect to a particular Entity. Forexample, the user request “Create a meeting with John at 1 p.m.tomorrow, please” may be presented as the following pattern: create ameeting @sys.any:subject @sys.date-time. Here, @sys.any:subject refersto an entity associated with a particular name, @sys.date-time refers toan entity associated with time and date, and the phrase “Create ameeting” refers to a predetermined action to be performed by DialogSystem Interface or Dialog System Engine with a certain mobileapplication, software application, or web service.

The procedures e) through h) mean that particular information in theform of text is provided with respect to a particular Entity. Forexample, the user request “Create a meeting with John at 1 p.m.tomorrow, please” may be presented as the following pattern: [Action]Text@[sys.date-time] Text. Here, @[sys.date-time] refers to an entityassociated with time and date, while the phrase “Create a meeting”refers to a predetermined action to be performed by Dialog SystemInterface 130 or Dialog System Engine 120 with a certain mobileapplication, software application, or web service. Element “Text” refersto content and not entity nor intent.

As mentioned above, a dialog system rule causes generation of alinguistic response and/or fulfilment response as an answer to a userrequest. One example of a linguistic response may include particularizedcontent deliverable as an audio message or displayable message.Fulfilment responses refer to particular processor-executableinstructions for one or more software applications, middleware,firmware, web services, and the like that cause implementation of aparticular action. Some examples of fulfilment responses may includescheduling an event in a calendar mobile application, writing andsending a text message or email, searching for content at a web searchservice, building a route in a navigational software application, and soforth. In certain embodiments, at least some linguistic responses and/orfulfilment responses can be configured by developers. In otherembodiments, at least some linguistic responses and/or fulfilmentresponses can be pre-configured and be available as default responses.

In certain additional embodiments, developers can provide examplerequests to illustrate intents and entities instead of providing“entities” and “intents.” In these embodiments, platform 110automatically determines, using machine-learning techniques, what“entities” and “intents” are implied in example user requests andcreates corresponding rules. For example, a developer may simply provideexample requests, such as “Play Beatles” and “I'd like to listen toMadonna,” and platform 110 will match “Beatles” and “Madonna” toexisting entities (system's or user's) and generate corresponding“[Action] @[Entity]” rules automatically.

Thus, developers can use platform interface 112 to generate a pluralityof dialog system rules specific to a particular application or industry.These pluralities of entities and intents form dialog system rules (alsoreferred to as dialog system elements) and enable custom Dialog SystemEngines to perform certain actions or generate certain outputs inresponse to a wide range of end user inputs.

FIG. 3 is a system 300 for expanding software applicationfunctionalities. Specifically, the system 300 may include a processor302 and a database 304 in communication with the processor 302. Thedatabase 304 may include computer-readable instructions for execution bythe processor 302.

The processor may be operable to receive a user request within asoftware application. In an example embodiment, the software applicationis associated with a user device. The software application may beenhanced with annotations that may include metadata. The enhancing ofthe software application with annotations may include integrating theannotations into a code associated with the software application. Inexample embodiments, the annotations may be pre-built annotations,annotations uploaded from a remote resource, annotations synchronizedwith an online resource, and so forth. Optionally, in an exampleembodiment, the processor 302 may be operable to load the metadata fromthe annotations to the dialog system. Therefore, the dialog system mayinclude metadata of all annotations, by which the software applicationassociated with the user device is enhanced. Upon receipt of the userrequest, the processor 302 may send at least a part of the user requestto a dialog system located on a remote device.

The processor 302 may be further operable to receive a dialog systemresponse to the user request from the dialog system. In an exampleembodiment, the dialog system response may include one or more of adisplayable message, an audio message, an incentive for the user toprovide a further user request, an instruction for the softwareapplication to perform a predetermined function, and so forth. Thedialog system response may include a callback URL to be accessed by thesoftware application to obtain a text of the dialog system response orat least one of the annotations associated with the dialog systemresponse.

Upon receipt of the dialog system response, the processor 302 mayidentify at least one of the annotations associated with the dialogsystem response to obtain an identified annotation. Based on theidentification, the processor 302 may invoke a code within the softwareapplication. The code may be associated with the identified annotation.In an example embodiment, the code may include an instruction for one ormore of the following: a further software application associated withthe user device, a server, a hosting service, a web service, a web site,and a cloud service.

FIG. 4 is a process flow diagram showing method 400 for expandingsoftware application functionalities at runtime, according to an exampleembodiment. The method may be performed by processing logic that maycomprise hardware (e.g., decision-making logic, dedicated logic,programmable logic, and microcode), software (such as software run on ageneral-purpose computer system or a dedicated machine), or acombination of both. In one example embodiment, the processing logicrefers to one or more components of a platform for creating andmaintaining custom Dialog System Engines. Notably, the below recitedsteps of method 400 may be implemented in an order different thandescribed and shown in FIG. 4. Moreover, method 400 may have additionalsteps not shown herein, but which can be evident for those skilled inthe art from the present disclosure. Method 400 may also have fewersteps than outlined below and shown in FIG. 4.

Method 400 commences at operation 410 with a user device receiving auser input within a software application running on the user device. Asdiscussed above, the software application is enhanced with one or moreannotations, which include metadata linking certain functions of thedialog system with functions of the software application. In general,the annotations can be used for various purposes. First, the annotationscan define types of requests that the software application should orcould handle by voice or text. More specifically, the annotations maydefine at least a type of the user request and the user request may beprocessed by the dialog system based on the type of the user request. Inthis example, the annotations can define entities, intents, and/oractions. Second, the annotations can define fulfilment of intents. Inthis example, the annotations could define code within the softwareapplication that needs to be invoked when an action is triggered withinthe dialog system. Third, the annotations can define runtime informationand user specific information that may be propagated to the dialogsystem and used in matching.

At operation 420, during runtime of the software application, the userdevice optionally loads information from one or more annotations to theDialog System employed on the platform (e.g., information is loaded tothe Dialog System Engine). The loaded information may include metadataconcerning intents, entities, actions, contexts, and/or fulfillmentsindicating which of the action should be taken when the intent isexecuted. In some embodiments, the loaded information may includecertain variables, parameters, and other data that can be involved inprocessing by a dialog system of the user request.

At operation 430, the user device transmits the user request to theplatform (i.e., to the Dialog System Engine).

At operation 440, the user device receives a dialog system response tothe user request from the Dialog System Engine. The dialog systemresponse can include a displayable or audio message, or an instructioncausing the user device to perform a certain action.

At operation 450, the user device identifies at least one of theannotations associated with the dialog system response (i.e., triggeredintent and action).

At operation 460, the user device invokes a code within the softwareapplication, which code is associated with the identified annotation.

In some embodiments, method 400 can be used for compilation of a dialogagent during design time and loading of the compiled dialog definitionto the dialog system. Alternatively, method 400 can be used forcompilation for embedded use only. In other words, the softwareapplication could define a dialog interface. When the softwareapplication is built, dialog definitions will be compiled and embeddedinto this application. The software application could run the dialoginterface with no connection to the dialog system (i.e., in a completelyembedded mode).

FIG. 5 is a process flow diagram showing method 500 for creating customDialog System Engines using platform 110 and for operating platform 110,according to an example embodiment. The method may be performed byprocessing logic that may comprise hardware (e.g., decision-makinglogic, dedicated logic, programmable logic, and microcode), software(such as software run on a general-purpose computer system or adedicated machine), or a combination of both. In one example embodiment,the processing logic refers to one or more components of platform 110.Notably, the below recited steps of method 500 may be implemented in anorder different than described and shown in FIG. 5. Moreover, method 500may have additional steps not shown herein, but which can be evident forthose skilled in the art from the present disclosure. Method 500 mayalso have fewer steps than outlined below and shown in FIG. 5.

At operation 505, a developer registers with platform 110. For theseends, the developer shall interact with platform interface 112. Theregistration may include creating a developer profile, which can bemaintained by platform 110. The developer profile may link (associate) acustom Dialog System Engine 120 of this developer and one or more DialogSystem Interfaces 130 deployed on the client side. More specifically, adeveloper profile may include multiple agents (such as custom dialogsystem engines) and each of them could be integrated into a clientapplication with the help of SDK. Accordingly, there can be multipleend-points for user input (Dialog System Interfaces). The communicationbetween Dialog System Engines 120 and Dialog System Interfaces 130 mayinclude Application Programming Interface (API) codes, rules forinteraction, destination addresses, and many more.

At optional operation 510, platform 110 receives from the developer oneor more entities and stores the same in a local database. In someembodiments, the entities are not received, but created by the developerusing web tools of platform interface 112. In yet other embodiments,entities are not created or received at all and, therefore, some dialogsystem engines, or dialog system engine rules may not have any entities.

At operation 515, platform 110 receives from the developer one or moreintents and stores the same at the local database. In some embodiments,the intents are not received, but created by the developer using toolsof platform interface 112. As described above, the intents areassociated with the entities, and together they form dialog systemelements (custom rules enabling Dialog System Engine 120 to generateresponses tailored for specific needs).

At operation 520, platform 110 associates one or more entities with oneor more intents to create (form) custom Dialog System Engine 120. DialogSystem Engine 120 is associated with one or more Dialog SystemInterfaces 130 of the developer.

Operations 505-520 illustrate a set-up process for the custom DialogSystem Engine 120, while the following operations 525-545 illustrate theoperation of custom Dialog System Engine 120. FIGS. 6-9 illustratevarious screenshots of platform interface 112, which show the process ofmanually creating entities (see FIG. 6), manually creating intents (seeFIGS. 7, 8), and a test console for testing dialog system elements (seeFIG. 9) by developers.

Once all dialog system elements of custom Dialog System Engine 120 arecreated, they are maintained as a backend service and enable any of theassociated Dialog System Interfaces 130 to provide the fullfunctionality of the Dialog System to users according to predeterminedsettings.

Referring back to FIG. 5, at operation 525, platform 110 receives a userrequest from an unidentified Dialog System Interface 130. The userrequest can be a voice input or text input. In some embodiments, DialogSystem Interface 130 can pre-process the user input, for example, byrecognizing spoken words and transforming the voice input into textinput. In other embodiments, however, no pre-processing is performed byDialog System Interface 130.

At operation 530, platform 110 processes the user request and identifiesDialog System Interface 130. For these ends, the user request can beaccompanied by an identifier (ID) at the time the user request is sentfrom Dialog System Interface 130 to platform 110. Generally, identifiershere may refer to (a) a user ID, which can be used to retrieve adeveloper's or end-user's profile; (b) a session ID, which can be usedto retrieve current dialog context between the given end user and theDialog System Engine; and (c) a set of keys/agents (i.e., custom dialogsystem IDs), which can be used to identify entities, intents, and otherdata associated with the user request.

At operation 535, based on the result of the identification at operation530, platform 110 activates Dialog System Engine 120 associated withidentified Dialog System Interface 130. At the same operation, platform110 may also retrieve or identify one or more dialog system elements(i.e., one or more entities and one or more intents) based on the resultof the identification at operation 530.

At operation 540, Dialog System Engine 120 processes the user requestusing identified dialog system elements (i.e., one or more entities andone or more intents) as retrieved at operation 535. Some examples ofdialog system processing are further described with reference to FIG. 4.

At operation 545, Dialog System Engine 120 generates a response andsends it to Dialog System Interface 130 associated with Dialog SystemEngine 120. Dialog System Interface 130 then displays and/or playbacksthe response to the end user depending on predetermined settings.

FIGS. 6-9 illustrate various screenshots of platform interface 112illustrating creating of dialog system rules.

FIG. 6 shows the process 600 of manual creating of entities via theplatform interface 112. In one example, one set of entities may includedefinitions of geographical locations including city names such as“Atlanta,” “Belfast,” “Boston,” “Chicago,” “Detroit,” and so forth. Inanother example, another set of entities may include time definitionssuch as “today,” “tomorrow,” “next week,” “next Monday,” “October 1,”and the like. It shall be clear that the context database 270 mayinclude multiple sets of entities each related to various activities.

As illustrated, the entity related to city may be associated withmultiple values, for example New York, Washington, Paris, and so forth.Each of the values may include a list of synonyms 610 and a singularreference value/canonical value 620. A developer can manage (add, edit,delete) the values of an entity and the list of synonyms 610 for eachvalue.

FIG. 7 shows the process 700 of manual creating of intents for a dialogsystem rule. Intents can be provided as multiple sets of linguisticelements such as words or phrases associated with particular actions.Some examples of intents associated with the action of playingmultimedia content may include “Play,” “Start playback,” “Activateplaying,” “Open,” and so forth. Additionally, the context database 270may also store multiple context words or phrases, such as “what about,”“and how is it in,” “and the same in,” “and what about,” and so forth,all associated with a particular dialog system rule. These contextphrases can facilitate interpretation of user inputs. In someembodiments, the context database 270 may also store pre-built lexicaldata including, but not limited to, synonyms (words, phrases). In yetmore embodiments, the context database 270 may also include pre-builtknowledge databases such as WordNet or the like.

FIG. 7 illustrates an intent including a multitude of user expressions710 identified automatically from exemplary phrases or specified by adeveloper, for example, by adding a new user expression using an Addcontrol 720. On fulfillment 730 of the rule associated with the intent,the system may provide a speech string of call a web service.Additionally, a developer can test the rule execution using an always-ontest console 740 by entering a test phrase.

FIG. 8 shows an example rule 800 for receiving a weather forecast. Asillustrated, the rule 800 includes multiple user expressions 810 thatcan be used by the user to request a weather forecast, for example,“what is the weather in,” “weather forecast for,” and so forth.Entities, such as city or date, in user expressions can be providedusing an alias 820 (or placeholder). Aliases can be referenced in action830 and in fulfillment 840 section. A developer can manage userexpressions 810, actions 830, and fulfillment 840 parameters.

New dialog system rules including intents and entities are stored in therule database 260. One way to create new dialog system rules is a manualinput of intents and entities, including specification of references toentities within intents. However, this approach may not be effective incertain instances and may have some disadvantages. For example, it canbe very time consuming for developers to specify various naturallanguage inputs to cover the wide range of inputs in which a certainuser request can be made. The present technology addresses this problemby platform 110 automatically creating dialog system rules in responseto receiving exemplary user requests from developers and/or analyzingrequests from multiple dialog system users. Embodiments of thisdisclosure provide for context database 270 to be automaticallypopulated with certain additional terms, keywords, phrases, and/orexpressions using machine-learning and/or pattern recognition techniqueswhen developers input exemplary user requests that potentially can beused by end users in the course of interaction with dialog system.

For example, developers may provide exemplary phrases such as “What isthe weather like in New York,” “What is the current temperature in LasVegas,” “What is the forecast for Washington D.C.,” “How it is inMoscow,” “Do I need my umbrella tomorrow,” and so forth. These and otherexemplary phrases can be provided via platform interface 112 and/orduring the interaction of developer or end user with dialog systemengines 120.

FIG. 9 shows a process 900 of testing dialog system elements bydevelopers. To test a rule, a developer can enter a test phrase in thetest console. The test phrase may be processed 910 by the system toidentify entities and intent in the test phrase. The result of theprocessing can be provided to the developer for review. Based on theprocessing, action associated with the intent may be provided as aresponse to the test phrase with reference values of the entities passedas a parameter 920. The developer can modify the intent according to thetest results.

Once all dialog system elements of dialog system engine 120 are created,they can be maintained as a backend service and enable any of theassociated dialog system interfaces 130 to provide the fullfunctionality of the dialog system to users according to predeterminedsettings.

FIG. 10 is a high-level block diagram illustrating an example userdevice 1000 suitable for implementing the methods described herein. Itis worth mentioning that all components of the user device may includelogic elements, hardware components, software (firmware) components,virtual components, or a combination thereof.

User device 1000 includes at least an integral part of one or more of avariety of types of devices and systems such as a general-purposecomputer, desktop computer, server, computer network, network service,cloud-computing service, and so forth. Further, all modules shown inFIG. 10 may be operatively coupled using any suitable wired, wireless,radio, electrical, or optical standards. As already outlined above, userdevice 1000 may refer to a smart phone, wireless telephone, computer,such as a tablet computer or desktop computer, infotainment system,in-vehicle computing device, and the like.

As shown in FIG. 10, user device 1000 includes the following hardwarecomponents: one or more processors 1002, memory 1004, one or morestorage devices 1006, one or more input modules 1008, one or more outputmodules 1010, network interface 1012, and optional geo locationdeterminer 1014. User device 1000 also includes the following softwareor virtual components: an operating system 1020, one or more software(mobile) applications 1030, and Dialog System Interface 130, which canbe a stand-alone software application or be integrated into one or moresoftware applications 1030. Dialog System Interface 130 may provide ahuman-centric interface for accessing and managing information asdiscussed herein, communicating with Dialog System Engine 120, andcommunicating with web resources/services 160. According to variousembodiments, Dialog System Interface 130 can be virtual, meaning itincludes annotations integrated into a code of software applications1030.

Processors 1002 are configured to implement functionality and/or processinstructions for execution within user device 1000. For example, theprocessors 1002 may process instructions stored in memory 1004 and/orinstructions stored on storage devices 1006. Such instructions mayinclude components of an operating system 1020 and software applications1030. The user device 1000 may also include one or more additionalcomponents not shown in FIG. 10, such as a housing, power supply,communication bus, and the like. These elements are omitted so as to notburden the description of present embodiments.

Memory 1004, according to one example embodiment, is configured to storeinformation within the user device 1000 during operation. Memory 1004may refer to a non-transitory computer-readable storage medium or acomputer-readable storage device. In some examples, memory 1004 is atemporary memory, meaning that a primary purpose of memory 1004 may notbe long-term storage. Memory 1004 may also refer to a volatile memory,meaning that memory 1004 does not maintain stored contents when memory1004 is not receiving power. Examples of volatile memories include RAM,DRAM, SRAM, and other forms of volatile memories known in the art. Insome examples, memory 1004 is used to store program instructions forexecution by the processors 1002. Memory 1004, in one exampleembodiment, is used by software (e.g., the operating system 1020) orDialog System Interface 130 executing on user device 1000 to temporarilystore information during program execution. One or more storage devices1006 can also include one or more transitory or non-transitorycomputer-readable storage media and/or computer-readable storagedevices. In some embodiments, storage devices 1006 may be configured tostore greater amounts of information than memory 1004. Storage devices1006 may further be configured for long-term storage of information. Insome examples, storage devices 1006 include non-volatile storageelements. Examples of such non-volatile storage elements includemagnetic hard discs, optical discs, solid-state discs, flash memories,forms of electrically programmable memories (EPROM) or electricallyerasable and programmable memories (EEPROM), and other forms ofnon-volatile memories known in the art.

Still referencing to FIG. 10, user device 1000 includes one or moreinput modules 1008. The input modules 1008 are configured to receiveuser inputs. Examples of input modules 1008 include a microphone,keyboard, keypad, mouse, trackball, touchscreen, touchpad, or any otherdevice capable of detecting an input from a user or other source in theform of speech, audio, or tactile actions, and relaying the input to theuser device 1000 or components thereof.

Output modules 1010, in some example embodiments, are configured toprovide output to users through visual or auditory channels. Outputmodules 1010 may include a video graphics adapter card, liquid crystaldisplay monitor, light emitting diode monitor, sound card, speaker, orany other device capable of generating output that may be intelligibleto a user.

User device 1000, in some embodiments, includes network interface 1012.Network interface 1012 can be utilized to communicate with externaldevices, servers, and networked systems via one or more communicationsnetworks such as one or more wired, wireless, or optical networksincluding, for example, the Internet, intranet, LAN, WAN, cellular phonenetworks (e.g., GSM communications network, packet switchingcommunications network, circuit switching communications network),Bluetooth radio, and an IEEE 802.11-based radio frequency network, amongothers. Network interface 1012 may be a network interface card, such asan Ethernet card, optical transceiver, radio frequency transceiver, orany other type of device that can send and receive information. Otherexamples of such network interfaces may include Bluetooth®, 3G, 4G, andWiFi® radios in mobile computing devices as well as Universal Serial Bus(USB).

User device 1000 may further include optional geo location determiner1014 for determining a current geographical location of the user device.Geo location determiner 1014 may utilize a number of different methodsfor determining geographical location including, for example, receivingand processing signals of Global Positioning Systems, GLONASS satellitenavigation systems, or the Galileo satellite navigation system;utilizing multilateration of radio signals between radio towers (basestations); or utilizing geolocation methods associated with InternetProtocol addresses, Media Access Control addresses, Radio-FrequencyIdentification, or other technologies.

Operating system 1020 may control one or more functionalities of userdevice 1000 or components thereof. For example, operating system 1020may interact with Dialog System Interface 130 and may further facilitateone or more interactions between software applications 1030 and one ormore of processors 1002, memory 1004, storage devices 1006, inputmodules 1008, and output modules 1010. As shown in FIG. 10, operatingsystem 1020 may interact with or be otherwise coupled to softwareapplications 1030, Dialog System Interface 130, and components thereof.In some embodiments, Dialog System Interface 130 can be included intothe operating system 1020 and/or software applications 1030. Notably,user device 1000 and its components such as the Dialog System Interface130, may also interact with one or more remote storage or computingresources including, for example, web resources, websites, socialnetworking websites, blogging websites, news feeds, email servers, webcalendars, event databases, ticket aggregators, map databases, points ofinterest databases, and so forth.

FIG. 11 is a high-level block diagram illustrating an example system1100 suitable for implementing the methods described herein. Inparticular, system 1100 is a server-based solution suitable for runningplatform 110.

Note that all components of system 1100 include logic elements, hardwarecomponents, software (firmware) components, virtual components, or acombination thereof. System 1100 may include, relate, or constitute anintegral part of one or more of a variety of types of devices andsystems such as a general-purpose computer, server, web server, networkservice, cloud-computing service, and so forth. Further, all modulesshown in FIG. 11 may be operatively coupled using any suitable wired,wireless, radio, electrical, or optical standards.

As shown in FIG. 11, system 1100 includes the following hardwarecomponents: one or more processors 1102, memory 1104, one or morestorage devices 1106, and network interface 1108. System 1100 alsoincludes the following software or virtual components: operating system1110, one or more software applications 1120, and developer interface1130 such as platform interface 112. The developer interface 1130 mayprovide a human-centric interface for accessing and managing informationas discussed herein.

In some embodiments, processors 1102 are configured to implementfunctionality and/or process instructions for execution within thesystem 1100. For example, processors 1102 may process instructionsstored in memory 1104 and/or instructions stored on storage devices1106. Such instructions may include components of operating system 1110,software applications 1120, and/or developer interface 1130.

Memory 1104, according to one example embodiment, is configured to storeinformation within system 1100 during operation. Memory 1104, in someexample embodiments, may refer to a non-transitory computer-readablestorage medium or a computer-readable storage device. In some examples,memory 1104 is a temporary memory, meaning that a primary purpose ofmemory 1104 may not be long-term storage. Memory 1104 may also refer toa volatile memory, meaning that memory 1104 does not maintain storedcontents when memory 1104 is not receiving power. Examples of volatilememories include RAM, DRAM, SRAM, and other forms of volatile memoriesknown in the art. In some examples, memory 1104 is used to store programinstructions for execution by the processors 1102. Memory 1104, in oneexample embodiment, is used to temporarily store information duringprogram execution.

One or more storage devices 1106 can also include one or more transitoryor non-transitory computer-readable storage media and/orcomputer-readable storage devices. In some embodiments, storage devices1106 may be configured to store greater amounts of information thanmemory 1104. Storage devices 1106 may further be configured forlong-term storage of information. In some examples, the storage devices1106 include non-volatile storage elements. Examples of suchnon-volatile storage elements include magnetic hard discs, opticaldiscs, solid-state discs, flash memories, forms of EPROM or EEPROM, andother forms of non-volatile memories known in the art. In one example,one or more storage devices 1106 can include the databases 260, 270, 280on FIG. 2. In other embodiments, one or more storage devices 1106 canstore and maintain user profiles and custom Dialog System Engines 120.

Still referencing to FIG. 11, system 1100 includes network interface1108. Network interface 1108 can be utilized to communicate withexternal devices, servers, and networked systems via one or morecommunications networks such as one or more wired, wireless, or opticalnetworks including, for example, the Internet, intranet, LAN, WAN,cellular phone networks (e.g. GSM communications network, packetswitching communications network, circuit switching communicationsnetwork), Bluetooth radio, and an IEEE 802.11-based radio frequencynetwork, among others. Network interface 1108 may be a network interfacecard, such as an Ethernet card, optical transceiver, radio frequencytransceiver, or any other type of device that can send and receiveinformation. Other examples of such network interfaces may includeBluetooth®, 3G, 4G, and WiFi® radios in mobile computing devices as wellas USB.

Operating system 1110 may control one or more functionalities of system1100 or components thereof. For example, operating system 1110 mayinteract with the developer interface 1130, and may further facilitateone or more interactions between software applications 1120 and one ormore of processors 1102, memory 1104, storage devices 1106, and/ornetwork interface 1108. As shown in FIG. 11, operating system 1110 mayinteract with or be otherwise coupled to the developer interface 1130and components thereof. Notably, system 1100 and its components may alsointeract with one or more remote storage or computing resources (such asthose shown on FIG. 1 as web resources/services 160) including, forexample, web resources, websites, social networking websites, bloggingwebsites, news feeds, email servers, web calendars, event databases,ticket aggregators, map databases, points of interest databases, and soforth. Software applications 1120, in essence, may provide functionalityto platform 110 and enable its operation. Alternatively, softwareapplications may be additions to platform 110.

Thus, methods and systems for expanding software application functionsby adding annotations to software code have been described. Althoughembodiments have been described with reference to specific exampleembodiments, it will be evident that various modifications and changescan be made to these example embodiments without departing from thebroader spirit and scope of the present application. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

What is claimed is:
 1. A method for expanding software applicationfunctionalities, the method being performed by at least one processorand a memory, the method comprising: receiving, via a particular dialogsystem interface of a particular software application running on a userdevice of a user, user input indicating a user request of the user,wherein each of a plurality of software applications installed on theuser device, including the particular software application, areassociated with a respective dialog system interface, and wherein eachrespective dialog system interface enables a dialog system agent tofulfill user requests with respect to their associated respectivesoftware application of the plurality of software applications;determining, by the particular dialog system interface, text inputcorresponding to the user request; loading fulfillment metadata of theparticular software application to the dialog system agent, wherein thefulfillment metadata of the particular software application includesmetadata to be used in fulfilling user requests with respect to theparticular software application; transmitting, via a first applicationprogramming interface (API) of the particular dialog system interface,the text input corresponding to the user request to the dialog systemagent; receiving, via the first API of the dialog system interface, adialog system response to the user request from the dialog system agent,wherein the dialog system response specifies a portion of thefulfillment metadata to be used in fulfilling the user request;identifying, based on the portion of the fulfillment metadata of theparticular software application specified by the dialog system responseand by the particular dialog system interface, a fulfillment action ofthe software application to be initiated based on the user request; andinitiating, by the particular dialog system interface of the particularsoftware application, the fulfillment action, wherein initiating thefulfillment action includes, at least, providing textual content to aseparate software application, of the plurality of software applicationsinstalled on the user device, via a second application programminginterface (API) of the particular dialog system interface.